Skip to content

Fix reject-inference.md Script Execution Output to match its own code - #567

Merged
yakew7 merged 1 commit into
yakew7:mainfrom
propcgamer20-png:fix/reject-inference-stale-output
Sep 10, 2026
Merged

Fix reject-inference.md Script Execution Output to match its own code#567
yakew7 merged 1 commit into
yakew7:mainfrom
propcgamer20-png:fix/reject-inference-stale-output

Conversation

@propcgamer20-png

Copy link
Copy Markdown
Collaborator

Closes #522.

The Script Execution Output block and the paragraph after it were pasted from an earlier version of the detection code and never regenerated. Running the file's own python block directly (seeded, byte-identical across repeated runs) gives:

                       Population AUC  Population Accuracy  Older Approval Rate  Younger Approval Rate  Age Fairness Gap
Naive (Approved Only)          0.9460               0.8851               0.5413                 0.5366            0.0047
IPW Reweighted                 0.9456               0.8813               0.5514                 0.5453            0.0062
Soft Parceled                  0.9238               0.8852               0.5413                 0.5369            0.0045

not the claimed 0.7812 -> 0.8345 AUC and 15.80 -> 2.30 point gap.

Why regenerating the prose (not just the table) was necessary

The old narrative ("15.80-point gap corrected down to 2.30") is not reachable with this simulation, not merely stale:

  • y_true is generated with no age term - older and younger applicants repay at the same ~55% rate.
  • Every model trains on credit_score and income_k only, both drawn independently of age_young.

So selection acting on age alone is ignorable for estimating P(Y | X); a correctly specified RandomForest recovers a near-parity score distribution with or without IPW, and there is no naive-model gap for the correction to close. IPW/parceling only bite under MNAR selection or when the outcome model omits a feature that drives selection - neither of which this minimal simulation creates.

Change

  • Output block replaced with the real deterministic figures.
  • Analysis rewritten to describe what the code actually shows: the disparity is entirely in the selection gate (older approval ~50%, younger ~35%; young applicants are 34.6% of the population but 27.1% of the approved pool), which no audit run on model scores or approved-only rows can see - i.e. exactly Why It Matters item 2.
  • Added a one-line caveat that a seeded RandomForestClassifier's low-order digits can shift across CPU/BLAS backends (same cross-platform effect discussed in model-drift.md cites a fabricated single-run gap/p-value for unfair.py (6.39%/p=0.348 instead of the real 7.16%/p=0.2564) #521).
  • Regenerated reject-inference.html, faircode/_explainers/reject-inference.md, and the llms-full.txt section.

No code in the fenced block was changed. check_em_dash.py, check_broken_links.py, check_generated_files_current.py all pass.

…closes yakew7#522)

The 'Script Execution Output' block and the analysis after it were pasted
from an earlier version of the detection code and never regenerated. The
extracted python block, run directly (seeded, byte-identical across runs),
produces near-zero age gaps and ~0.92-0.95 population AUC across all three
models - not the 15.80 -> 2.30 point gap and 0.7812 -> 0.8345 AUC the doc
claimed.

That is not just a stale number: it is structurally unreachable with this
simulation. y_true carries no age term and the models train on
credit_score and income_k only, both drawn independently of age_young, so
selection acting on age alone is ignorable for estimating P(Y|X) and a
RandomForest recovers a near-parity score distribution with or without
IPW - there is no naive-model gap for the correction to close.

Replaced the output block with the real deterministic figures and
rewrote the analysis to describe what the code actually demonstrates: the
disparity lives entirely in the selection gate (older approval ~50%,
younger ~35%), invisible to any audit run on model scores or on the
approved-only rows - which is exactly 'Why It Matters' item 2. Added a
note that the low-order digits of a seeded RF can shift across
CPU/BLAS backends. Regenerated the HTML, package mirror, and llms-full.txt
section.
@vercel

vercel Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@propcgamer20-png is attempting to deploy a commit to the yashkewlani2020-gmailcom's projects Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions

Copy link
Copy Markdown

@yakew7 @Shreyash0712 - new PR touching a path you own, please review.

@yakew7
yakew7 merged commit cacb640 into yakew7:main Sep 10, 2026
18 of 19 checks passed
yakew7 added a commit that referenced this pull request Sep 10, 2026
Documents the newest explainer (#565), the CODEOWNERS grant to
@propcgamer20-png, 8 doc/code fixes (#557-564, #566, #567), and the
two direct fixes made during this review: the README.md regression
from #557's broad substring replacement, and the recurred
llms.txt/ROADMAP.md explainer-count drift after #565.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
yakew7 added a commit that referenced this pull request Sep 10, 2026
- Moved @propcgamer20-png into Core contributors (43 merged PRs, now
  code owner for /Makefile and /scripts/), with their bio extended to
  cover the #558-567 batch.
- Added @slsgzs-cloud (3 merged PRs, first merged 2026-09-08), missing
  from this file despite #503/#504 already being merged.
- Snapshot bumped to 2026-09-10, covering through PR #567.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reject-inference.md's "Script Execution Output" doesn't match its own code

2 participants